Advanced Lane Finding Project Pipeline (image level)¶
Advanced Lane Finding Project
The goals / steps of this project are the following:
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
In [1]:
# import libraries necessary
import os
import numpy as np
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import pickle
import glob
%matplotlib inline
1. Read undistortion parameter¶
- By using the unsitortion matrix which was from camera calibration. With 'pickle.load' function, I readed calibration matrix 'mtx' and distortion matrix 'dist' from camera calibaration.
In [2]:
#load distortion matrix from camera_cal
with open('./camera_cal/wide_dist_pickle.p',mode='rb') as f:
dist_pickle = pickle.load(f)
mtx = dist_pickle['mtx']
dist = dist_pickle["dist"]
print(mtx)
print(dist)
print('loaded mtx matrix and distortion matrix from undistortion')
2. Pipeline(single images)¶
Calibration function¶
- By using the distortion matrix and calibaration matrix , I defined a function used 'cv2.undistort' function.
In [3]:
#read image and change into BGR
img = cv2.imread('test_images/test5.jpg')
image = mpimg.imread('test_images/test5.jpg')
In [4]:
def undistort(img):
return cv2.undistort(img, mtx, dist, None, mtx)
In [5]:
plt.imshow(undistort(image))
Out[5]:
Binarize function¶
- In order to extract useful features from image, I have to do some preprocess step. In this project, since the line in left side is always yellow, so the color space transform was used. HLS color space method was used in this pipeline. After that, I need edges of the image, so I used sobel operation in x direction and y direction. But I used more in x direction since the line is horizontal. So implemented sobel operator in x direction is more efficiency. I also used lightness 'L' in HLS color space. By tuning the threshold in s channel, sx channel and lightness channel, then final output returned.
- The pixels accorded to the thresholds considered as 'hot'pixel. 'hot' pixel consisted a new binary image. This new image contains HLS information and edges information. By confirmed the final output figure whiche is named 'binarize' below, It extracts lines well. Next , I tried to give it a ROI to filtering the useless pixels.
In [6]:
# Define a function that thresholds the S-channel of HLS
# Use exclusive lower bound (>) and inclusive upper (<=)
def binarize(img, s_thresh=(120, 255), sx_thresh=(20, 255),l_thresh=(40,255)):
img = np.copy(img)
# Convert to HLS color space and separate the V channel
hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS).astype(np.float)
#h_channel = hls[:,:,0]
l_channel = hls[:,:,1]
s_channel = hls[:,:,2]
# Sobel x
# sobelx = abs_sobel_thresh(img, orient='x', sobel_kernel=3, thresh=(0, 255))
# l_channel_col=np.dstack((l_channel,l_channel, l_channel))
sobelx = cv2.Sobel(l_channel, cv2.CV_64F, 1, 0) # Take the derivative in x
abs_sobelx = np.absolute(sobelx) # Absolute x derivative to accentuate lines away from horizontal
scaled_sobel = np.uint8(255*abs_sobelx/np.max(abs_sobelx))
# Threshold x gradient
sxbinary = np.zeros_like(scaled_sobel)
sxbinary[(scaled_sobel >= sx_thresh[0]) & (scaled_sobel <= sx_thresh[1])] = 1
# Threshold saturation channel
s_binary = np.zeros_like(s_channel)
s_binary[(s_channel >= s_thresh[0]) & (s_channel <= s_thresh[1])] = 1
# Threshold lightness
l_binary = np.zeros_like(l_channel)
l_binary[(l_channel >= l_thresh[0]) & (l_channel <= l_thresh[1])] = 1
channels = 255*np.dstack(( l_binary, sxbinary, s_binary)).astype('uint8')
R = img[:,:,0]
thresh = (200, 255)
R_binary = np.zeros_like(R)
R_binary[(R > thresh[0]) & (R <= thresh[1])] = 1
binary = np.zeros_like(sxbinary)
binary[((l_binary == 1) & (s_binary == 1) | (R_binary == 1)&(sxbinary==1))] = 1
binary = 255*np.dstack((binary,binary,binary)).astype('uint8')
return binary,channels
binarize_image,channels = binarize(img)
# Plot the result
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(image)
ax1.set_title('Original Image', fontsize=50)
ax2.imshow(binarize_image, cmap='gray')
ax2.set_title('Binarize', fontsize=50)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
In [7]:
img = cv2.imread('test_images/test5.jpg')
shape = img.shape
binary,channels = binarize(img)
# Plot the result
f, (ax1,ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(binary)
ax1.set_title('Binary', fontsize=40)
ax2.imshow(channels)
ax2.set_title('Channels', fontsize=40)
Out[7]:
Perspective¶
- The code for my perspective transform includes a function called
warp(). This function receives a image and based on the predefined source points ('scr') and destination points ('dst'). Then output is warped image based on 'src','dst'perspective relationship.
In [8]:
# the choosen source point was selected as below
plt.imshow(image)
plt.plot(560,470,'.')
plt.plot(180,720,'.')
plt.plot(1180,720,'.')
plt.plot(750,470,'.')
Out[8]:
- The corner in warp() function set as below.
corners = np.float32([[515,470],[180,720], [1180,730],[770,470]])
- The offset defined as below
offset = 150
- The destination function was setted as below. With this destination function , I can extract the line's function more easily in x-y coordinate.
dst = np.float32([[150, 0], [150, 1280],[570, 1280],[570,0]])
- By using the 'cv2.getPerspectiveTransfrom(src,dst)'function and 'cv2.warpPersperctive) function , finally generats warped image.
- Warped image size is (720, 1280, 3)
In [9]:
corners = np.float32([[570,470],[165,720], [1135,720],[740,470]])
def warp(img,tobird=True):
offset = 150
img_size = (img.shape[1], img.shape[0])
src = np.float32(
[corners[0],
corners[1],
corners[2],
corners[3]])
print(src)
dst = np.float32([
[offset, 0],
[offset, img_size[1]],
[img_size[0] - offset, img_size[1]],
[img_size[0] - offset,0]])
if tobird:
M = cv2.getPerspectiveTransform(src, dst)
else:
M = cv2.getPerspectiveTransform(dst,src)
warped = cv2.warpPerspective(img, M, img_size , flags=cv2.INTER_LINEAR)
return warped
In [10]:
img=plt.imread('test_images/test5.jpg')
img = cv2.undistort(img, mtx, dist, None, mtx)
imshape = img.shape
corner_tuples=[]
for ind,c in enumerate(corners):
corner_tuples.append(tuple(corners[ind]))
cv2.line(img, corner_tuples[0], corner_tuples[1], color=[255,0,0], thickness=2)
cv2.line(img, corner_tuples[1], corner_tuples[2], color=[255,0,0], thickness=2)
cv2.line(img, corner_tuples[2], corner_tuples[3], color=[255,0,0], thickness=2)
cv2.line(img, corner_tuples[3], corner_tuples[0], color=[255,0,0], thickness=2)
warped= warp(img)
# Plot the result
f, (ax1,ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(img)
ax1.set_title('Original', fontsize=40)
ax2.imshow(warped)
ax2.set_title('Warped', fontsize=40)
Out[10]:
In [11]:
def region_of_interest(img):
"""
Applies an image mask.
Only keeps the region of the image defined by the polygon
formed from `vertices`. The rest of the image is set to black.
"""
shape = img.shape
vertices = np.array([[(0,0),(shape[1],0),(shape[1],0),(6*shape[1]/2,shape[0]),
(shape[1]/13,shape[0]), (0,0)]],dtype=np.int32)
mask = np.zeros_like(img)
#defining a 3 channel or 1 channel color to fill the mask with depending on the input image
if len(img.shape) > 2:
channel_count = img.shape[2] # i.e. 3 or 4 depending on your image
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
#filling pixels inside the polygon defined by "vertices" with the fill color
cv2.fillPoly(mask, vertices, ignore_mask_color)
#returning the image only where mask pixels are nonzero
masked_image = cv2.bitwise_and(img, mask)
return masked_image
def warp_pipeline(img):
undist = undistort(img)
result = warp(undist)
result = region_of_interest(result)
return result
def warp_binarize_pipeline(img):
undist = undistort(img)
binary,_ = binarize(undist)
result = warp(binary)
result = region_of_interest(result)
return result
warp_roi = warp_pipeline(img)
warp_binary_roi = warp_binarize_pipeline(img)
# Plot the result
f, (ax1,ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(warp_roi)
ax1.set_title('Warped ROI', fontsize=40)
ax2.imshow(warp_binary_roi)
ax2.set_title('Warped binary ROI', fontsize=40)
Out[11]:
In [12]:
# now back to the test image
img=plt.imread('test_images/test5.jpg')
warped = warp_pipeline(img)
warped_binary = warp_binarize_pipeline(img)
# Plot the result
f, (ax1,ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(warped)
ax1.set_title('Warped ROI', fontsize=40)
ax2.imshow(warped_binary)
ax2.set_title('Warped binary ROI', fontsize=40)
Out[12]:
- After process the image , I need to inverse the perspective to origin image, so I define a perspective inverse function 'Minv(img)' here.
In [13]:
def Minv(img):
corners = np.float32([[570,470],[170,720], [1145,720],[735,470]])
offset = 150
img_size = (img.shape[1], img.shape[0])
src = np.float32(
[corners[0],
corners[1],
corners[2],
corners[3]])
dst = np.float32([
[offset, 0],
[offset, img_size[1]],
[img_size[0] - offset, img_size[1]],
[img_size[0] - offset,0]])
Minv = cv2.getPerspectiveTransform(dst, src)
return Minv
Extract_lane in image¶
- main process of this pipeline. It gets input of image , and gives a output of mask image with extract lane.
- first of all is extract feature binary pixel image by using the 'warped_binarzie_pipeline'.
- Take a histogram of pixel to find which part of x is in left line and which part is right line.
- saperate lane by using the histogram and using sliding windows find left line and right line pixel's position indices.
- by using the indices, I fit a polynominal function to the line's x and y with 2nd order polynominal(y =ax^2+bx+c). Then I got a constant coefficiency a,b,c of left_fit and right_fit. With this functions, I can extract where the line is , where the left line is , where the right line is and the curvature and postion from center point. Position from center point is calculate by the nearest X postion by using 2nd order fitted polynominal formular.
left_fit = np.polyfit(lefty, leftx, 2)
right_fit = np.polyfit(righty, rightx, 2)
- Since I got the formular of the line, I perspect the pixel coordinate to the real-world coordinate. The perspective formular used formular as below.
ym_per_pix = 3.0/72.0
xm_per_pix = 3.7/660.0
- After extract line formular, I used ' cv2.fill poly ' to visualize the lane and add curvature and position from center position.
In [14]:
#main process to extract_lane from image
def extract_lane(img):
#process the image with warp_binarize_pipeline
warped_binary = warp_binarize_pipeline(img)
#convert 'warped_binary' to gray. This step is in order to reduce the scale of the image.
binary_warped = cv2.cvtColor(warped_binary,cv2.COLOR_RGB2GRAY)
# Take a histogram of the bottom half of the image
histogram = np.sum(binary_warped[binary_warped.shape[0]//2:,:], axis=0)
# Create an output image to draw on and visualize the result
out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
# Find the peak of the left and right halves of the histogram
# These will be the starting point for the left and right lines
midpoint = np.int(histogram.shape[0]//2)
leftx_base = np.argmax(histogram[:midpoint])
rightx_base = np.argmax(histogram[midpoint:]) + midpoint
# Choose the number of sliding windows
nwindows = 9
# Set height of windows
window_height = np.int(binary_warped.shape[0]//nwindows)
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Current positions to be updated for each window
leftx_current = leftx_base
rightx_current = rightx_base
# Set the width of the windows +/- margin
margin = 100
# Set minimum number of pixels found to recenter window
minpix = 50
# Create empty lists to receive left and right lane pixel indices
left_lane_inds = []
right_lane_inds = []
# Step through the windows one by one
for window in range(nwindows):
# Identify window boundaries in x and y (and right and left)
win_y_low = binary_warped.shape[0] - (window+1)*window_height
win_y_high = binary_warped.shape[0] - window*window_height
win_xleft_low = leftx_current - margin
win_xleft_high = leftx_current + margin
win_xright_low = rightx_current - margin
win_xright_high = rightx_current + margin
# Draw the windows on the visualization image
cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),
(0,255,0), 2)
cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),
(0,255,0), 2)
# Identify the nonzero pixels in x and y within the window
good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
(nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]
good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
(nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]
# Append these indices to the lists
left_lane_inds.append(good_left_inds)
right_lane_inds.append(good_right_inds)
# If you found > minpix pixels, recenter next window on their mean position
if len(good_left_inds) > minpix:
leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
if len(good_right_inds) > minpix:
rightx_current = np.int(np.mean(nonzerox[good_right_inds]))
# Concatenate the arrays of indices
left_lane_inds = np.concatenate(left_lane_inds)
right_lane_inds = np.concatenate(right_lane_inds)
# Extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
left_fit = np.polyfit(lefty, leftx, 2)
right_fit = np.polyfit(righty, rightx, 2)
##################################################################################
ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
##################################################################################
# Create an image to draw on and an image to show the selection window
out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
window_img = np.zeros_like(out_img)
# Color in left and right line pixels
out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
# Generate a polygon to illustrate the search window area
# And recast the x and y points into usable format for cv2.fillPoly()
left_line_window1 = np.array([np.transpose(np.vstack([left_fitx-margin, ploty]))])
left_line_window2 = np.array([np.flipud(np.transpose(np.vstack([left_fitx+margin,
ploty])))])
left_line_pts = np.hstack((left_line_window1, left_line_window2))
right_line_window1 = np.array([np.transpose(np.vstack([right_fitx-margin, ploty]))])
right_line_window2 = np.array([np.flipud(np.transpose(np.vstack([right_fitx+margin,
ploty])))])
right_line_pts = np.hstack((right_line_window1, right_line_window2))
# Draw the lane onto the warped blank image
cv2.fillPoly(window_img, np.int_([left_line_pts]), (0,255, 0))
cv2.fillPoly(window_img, np.int_([right_line_pts]), (0,255, 0))
#################################################################################
y_eval = np.max(ploty)
left_curverad = ((1 + (2*left_fit[0]*y_eval + left_fit[1])**2)**1.5) / np.absolute(2*left_fit[0])
right_curverad = ((1 + (2*right_fit[0]*y_eval + right_fit[1])**2)**1.5) / np.absolute(2*right_fit[0])
#################################################################################
ym_per_pix = 3.0/72.0 # meters per pixel in y dimension
xm_per_pix = 3.7/660.0 # meters per pixel in x dimension
y_eval = 700
midx = 650
#genete the curvature based on real world transfrom with pexel relationship.
y1 = (2*left_fit[0]*y_eval + left_fit[1])*xm_per_pix/ym_per_pix
y2 = 2*left_fit[0]*xm_per_pix/(ym_per_pix*ym_per_pix)
#curvature calculate based on extract left line .
curvature = ((1 + y1*y1)**(1.5))/np.absolute(y2)
#############################################################################################
#calculate the position from center by using the x position in left line and right line.
#Using the 2nd order polynomial function to generate x left pixel postion and x right pixel position.
x_left_pix =left_fit[0]*(y_eval**2) + left_fit[1]*y_eval +left_fit[2]
x_right_pix = right_fit[0]*(y_eval**2) + right_fit[1]*y_eval +right_fit[2]
pos_from_center = ((x_left_pix + x_right_pix)/2.0 - midx) * xm_per_pix
##############################################################################################
warped = binary_warped
warp_zero = np.zeros_like(warped).astype(np.uint8)
color_warp = np.dstack((warp_zero, warp_zero, warp_zero))
# Recast the x and y points into usable format for cv2.fillPoly()
pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
pts = np.hstack((pts_left, pts_right))
# Draw the lane onto the warped blank image
cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))
# Warp the blank back to original image space using inverse perspective matrix (Minv)
newwarp = cv2.warpPerspective(color_warp, Minv(img), (image.shape[1], image.shape[0]))
# Combine the result with the original image
result = cv2.addWeighted(undistort(img), 1, newwarp, 0.3, 0)
###############################################################################################
#add pos_from_center and curvature string on the image.
font = cv2.FONT_HERSHEY_SIMPLEX
str1=str('position from center is : '+ str(pos_from_center) + 'cm')
cv2.putText(result,str1,(380,600),font,1,(0,0,255),2,cv2.LINE_AA)
str2 = str('radius of curvature: '+str(curvature*0.001)+'km')
cv2.putText(result,str2,(430,670), font, 1,(0,0,255),2,cv2.LINE_AA)
return result
In [15]:
#test the extract_lane fucntion with test image.
plt.imshow(extract_lane(image))
Out[15]:
In [16]:
print('Extract_lane main process image leve finished ')
Advanced Lane Finding Project Pipeline (video level)¶
- In video level test, I use the functions used in pipline (image level)
- By using the 'VideoFileClip' library to extract image in video and process the image one by one.
- 'VideoFileClip' library read video, and extract image to main process, and main process gives a output to 'VideoFileClip'.
- Finally, the every image process by main function will be regathered with a new video named 'Ming Lin_processed_project_video.mp4'
In [17]:
# main process used in VideoFileClip defined.
def main(img):
result = extract_lane(img)
return result
In [18]:
import pickle
import glob
import scipy
from scipy import signal
from collections import deque
import imageio
imageio.plugins.ffmpeg.download()
# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
In [19]:
output = './Ming Lin_processed_project_video2.mp4'
clip = VideoFileClip("./project_video.mp4")
In [20]:
out_clip = clip.fl_image(main)
%time out_clip.write_videofile(output, audio=False)
Discussion¶
problems/issues faced¶
- In warp and binarize part, how to generate the useful features from image.
- The warp vertices and binarize threshold were hard to tune. There are no specific requirement. So I don't know if the result is good enough.
- There are still so many hyperparameter need human to tuning. I don't know if it can still word well in other scenario.
what could be improved about algorithm/pipeline¶
- The sliding window algorithm was not used in here. If use sliding window algorith, I think the process will run faster than now.
- I think If machine learning algorithm in binarize part(feature pixel), maybe it can help us to find points, edges, rectangle, circle, inner relationship of each pixel and each image. Maybe the CNN and RNN can be used in here to extract information from flowing image stream.
what hypothetical cases would cause pipeline to fail¶
- This project scenario are all in high way. The high way has good road condition, so the algorithm gets little noise information from road to extract lane.
- The scenario is daytime. if the environment becomes night time, then it will get more noise pixel because of different kinds of lights like car lamp, street lamp and reflections from everywhere.
- In real vehicle, whether the processor can processes there large amount of data or not is one of the critical problems.